Pandas Profiling Report

Dataset statistics

Number of variables	12
Number of observations	64150
Missing cells	0
Missing cells (%)	0.0%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	5.9 MiB
Average record size in memory	96.0 B

Variable types

DateTime	1
Numeric	11

Warnings

`GAES_GEN` has 42114 (65.6%) zeros	Zeros
`GAES_PUMP` has 44839 (69.9%) zeros	Zeros

Reproduction

Analysis started	2021-05-03 11:59:33.243216
Analysis finished	2021-05-03 11:59:58.022128
Duration	24.78 seconds
Software version	pandas-profiling v2.11.0
Download configuration	config.yaml

datereport
Date

Distinct	64149
Distinct (%)	> 99.9%
Missing	0
Missing (%)	0.0%
Memory size	501.3 KiB

Minimum	2014-01-01 01:00:00
Maximum	2021-04-28 07:00:00

Histogram

Histogram with fixed size bins (bins=50)

AES
Real number (ℝ_≥0)

Distinct	5342
Distinct (%)	8.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	9561.336352
Minimum	6014
Maximum	12724
Zeros	0
Zeros (%)	0.0%
Memory size	501.3 KiB

Quantile statistics

Minimum	6014
5-th percentile	7448
Q1	8485
median	9643
Q3	10591
95-th percentile	11553
Maximum	12724
Range	6710
Interquartile range (IQR)	2106

Descriptive statistics

Standard deviation	1283.710613
Coefficient of variation (CV)	0.1342605851
Kurtosis	-0.9843520169
Mean	9561.336352
Median Absolute Deviation (MAD)	1055
Skewness	-0.1627758884
Sum	613359727
Variance	1647912.938
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
10542	50	0.1%
9533	49	0.1%
10737	49	0.1%
10756	47	0.1%
10531	47	0.1%
10754	46	0.1%
10538	46	0.1%
10747	45	0.1%
10973	44	0.1%
10551	44	0.1%
Other values (5332)	63683	99.3%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
6014	1	< 0.1%
6249	1	< 0.1%
6308	1	< 0.1%
6314	1	< 0.1%
6341	1	< 0.1%

Value	Count	Frequency (%)
12724	2	< 0.1%
12712	1	< 0.1%
12641	2	< 0.1%
12639	1	< 0.1%
12638	2	< 0.1%

TEC
Real number (ℝ_≥0)

Distinct	2495
Distinct (%)	3.9%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1125.80781
Minimum	349
Maximum	2910
Zeros	0
Zeros (%)	0.0%
Memory size	501.3 KiB

Quantile statistics

Minimum	349
5-th percentile	471
Q1	606
median	1034
Q3	1390
95-th percentile	2356
Maximum	2910
Range	2561
Interquartile range (IQR)	784

Descriptive statistics

Standard deviation	597.2644043
Coefficient of variation (CV)	0.5305207506
Kurtosis	-0.1475933883
Mean	1125.80781
Median Absolute Deviation (MAD)	419
Skewness	0.885943425
Sum	72220571
Variance	356724.7686
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
568	195	0.3%
537	164	0.3%
538	158	0.2%
540	157	0.2%
546	150	0.2%
528	139	0.2%
539	139	0.2%
536	138	0.2%
634	137	0.2%
533	134	0.2%
Other values (2485)	62639	97.6%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
349	1	< 0.1%
350	2	< 0.1%
351	1	< 0.1%
352	1	< 0.1%
353	3	< 0.1%

Value	Count	Frequency (%)
2910	2	< 0.1%
2904	1	< 0.1%
2888	1	< 0.1%
2879	1	< 0.1%
2872	1	< 0.1%

VDE
Real number (ℝ)

Distinct	2874
Distinct (%)	4.5%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	343.0272954
Minimum	-1
Maximum	4383
Zeros	105
Zeros (%)	0.2%
Memory size	501.3 KiB

Quantile statistics

Minimum	-1
5-th percentile	19
Q1	86
median	191
Q3	342
95-th percentile	1366
Maximum	4383
Range	4384
Interquartile range (IQR)	256

Descriptive statistics

Standard deviation	491.5684643
Coefficient of variation (CV)	1.433030172
Kurtosis	11.88432141
Mean	343.0272954
Median Absolute Deviation (MAD)	116
Skewness	3.203941913
Sum	22005201
Variance	241639.5551
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
36	236	0.4%
49	234	0.4%
20	232	0.4%
21	227	0.4%
34	215	0.3%
44	215	0.3%
59	214	0.3%
25	213	0.3%
33	212	0.3%
54	212	0.3%
Other values (2864)	61940	96.6%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-1	48	0.1%
0	105	0.2%
1	104	0.2%
2	123	0.2%
3	129	0.2%

Value	Count	Frequency (%)
4383	1	< 0.1%
4026	1	< 0.1%
3924	1	< 0.1%
3882	1	< 0.1%
3871	1	< 0.1%

TES
Real number (ℝ_≥0)

Distinct	9737
Distinct (%)	15.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	6144.674778
Minimum	2064
Maximum	17967
Zeros	0
Zeros (%)	0.0%
Memory size	501.3 KiB

Quantile statistics

Minimum	2064
5-th percentile	3358
Q1	4637
median	5801
Q3	7266
95-th percentile	10234
Maximum	17967
Range	15903
Interquartile range (IQR)	2629

Descriptive statistics

Standard deviation	2113.558691
Coefficient of variation (CV)	0.3439659164
Kurtosis	1.648975109
Mean	6144.674778
Median Absolute Deviation (MAD)	1283
Skewness	1.026298519
Sum	394180887
Variance	4467130.339
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
5682	28	< 0.1%
5121	28	< 0.1%
5429	27	< 0.1%
5608	26	< 0.1%
5061	25	< 0.1%
5017	25	< 0.1%
5400	25	< 0.1%
5058	24	< 0.1%
5312	24	< 0.1%
5560	24	< 0.1%
Other values (9727)	63894	99.6%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
2064	1	< 0.1%
2069	1	< 0.1%
2070	1	< 0.1%
2081	1	< 0.1%
2101	1	< 0.1%

Value	Count	Frequency (%)
17967	1	< 0.1%
17955	1	< 0.1%
17870	1	< 0.1%
17690	1	< 0.1%
17658	1	< 0.1%

GES
Real number (ℝ_≥0)

Distinct	2940
Distinct (%)	4.6%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	843.2490725
Minimum	40
Maximum	3695
Zeros	0
Zeros (%)	0.0%
Memory size	501.3 KiB

Quantile statistics

Minimum	40
5-th percentile	71
Q1	324
median	715
Q3	1237
95-th percentile	2064
Maximum	3695
Range	3655
Interquartile range (IQR)	913

Descriptive statistics

Standard deviation	630.4036642
Coefficient of variation (CV)	0.7475889209
Kurtosis	0.2446437739
Mean	843.2490725
Median Absolute Deviation (MAD)	436
Skewness	0.8657316114
Sum	54094428
Variance	397408.7798
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
66	268	0.4%
68	196	0.3%
64	190	0.3%
60	189	0.3%
69	186	0.3%
65	175	0.3%
67	172	0.3%
70	137	0.2%
72	133	0.2%
75	130	0.2%
Other values (2930)	62374	97.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
40	5	< 0.1%
41	11	< 0.1%
42	29	< 0.1%
43	62	0.1%
44	61	0.1%

Value	Count	Frequency (%)
3695	2	< 0.1%
3617	1	< 0.1%
3570	1	< 0.1%
3564	1	< 0.1%
3506	1	< 0.1%

GAES_GEN
Real number (ℝ_≥0)

ZEROS

Distinct	1165
Distinct (%)	1.8%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	162.7360561
Minimum	0
Maximum	1513
Zeros	42114
Zeros (%)	65.6%
Memory size	501.3 KiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	321
95-th percentile	786
Maximum	1513
Range	1513
Interquartile range (IQR)	321

Descriptive statistics

Standard deviation	270.7741951
Coefficient of variation (CV)	1.663885691
Kurtosis	2.656801204
Mean	162.7360561
Median Absolute Deviation (MAD)	0
Skewness	1.775248653
Sum	10439518
Variance	73318.66475
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	42114	65.6%
320	1839	2.9%
323	1784	2.8%
324	1610	2.5%
322	1584	2.5%
321	1397	2.2%
325	822	1.3%
151	408	0.6%
644	379	0.6%
319	352	0.5%
Other values (1155)	11861	18.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	42114	65.6%
2	1	< 0.1%
3	2	< 0.1%
4	7	< 0.1%
6	4	< 0.1%

Value	Count	Frequency (%)
1513	1	< 0.1%
1510	1	< 0.1%
1509	1	< 0.1%
1473	1	< 0.1%
1463	1	< 0.1%

CONSUMPTION
Real number (ℝ_≥0)

Distinct	12449
Distinct (%)	19.4%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	17129.04971
Minimum	10905
Maximum	30727
Zeros	0
Zeros (%)	0.0%
Memory size	501.3 KiB

Quantile statistics

Minimum	10905
5-th percentile	12711
Q1	15122
median	16671.5
Q3	19174
95-th percentile	22177
Maximum	30727
Range	19822
Interquartile range (IQR)	4052

Descriptive statistics

Standard deviation	2911.994458
Coefficient of variation (CV)	0.1700032697
Kurtosis	0.07259270441
Mean	17129.04971
Median Absolute Deviation (MAD)	1940.5
Skewness	0.5165082332
Sum	1098828539
Variance	8479711.721
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
15810	23	< 0.1%
15989	22	< 0.1%
15859	22	< 0.1%
15936	21	< 0.1%
15582	21	< 0.1%
16044	21	< 0.1%
16527	20	< 0.1%
15454	20	< 0.1%
15392	20	< 0.1%
16370	20	< 0.1%
Other values (12439)	63940	99.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
10905	1	< 0.1%
10907	1	< 0.1%
11036	1	< 0.1%
11152	1	< 0.1%
11157	1	< 0.1%

Value	Count	Frequency (%)
30727	2	< 0.1%
30490	1	< 0.1%
30351	1	< 0.1%
29916	1	< 0.1%
29904	1	< 0.1%

GAES_PUMP
Real number (ℝ)

ZEROS

Distinct	1076
Distinct (%)	1.7%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	-221.4053468
Minimum	-1400
Maximum	0
Zeros	44839
Zeros (%)	69.9%
Memory size	501.3 KiB

Quantile statistics

Minimum	-1400
5-th percentile	-1149
Q1	-393
median	0
Q3	0
95-th percentile	0
Maximum	0
Range	1400
Interquartile range (IQR)	393

Descriptive statistics

Standard deviation	394.3763332
Coefficient of variation (CV)	-1.781241234
Kurtosis	0.6762130996
Mean	-221.4053468
Median Absolute Deviation (MAD)	0
Skewness	-1.498530858
Sum	-14203153
Variance	155532.6922
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	44839	69.9%
-92	304	0.5%
-707	239	0.4%
-87	220	0.3%
-703	212	0.3%
-488	194	0.3%
-8	189	0.3%
-219	183	0.3%
-1156	150	0.2%
-1160	141	0.2%
Other values (1066)	17479	27.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-1400	1	< 0.1%
-1398	1	< 0.1%
-1392	1	< 0.1%
-1391	2	< 0.1%
-1390	1	< 0.1%

Value	Count	Frequency (%)
0	44839	69.9%
-1	84	0.1%
-2	17	< 0.1%
-3	100	0.2%
-4	82	0.1%

UK_BLR_RUS
Real number (ℝ)

Distinct	2296
Distinct (%)	3.6%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	-17.59382697
Minimum	-1185
Maximum	1928
Zeros	195
Zeros (%)	0.3%
Memory size	501.3 KiB

Quantile statistics

Minimum	-1185
5-th percentile	-414
Q1	-123
median	-36
Q3	52
95-th percentile	467.55
Maximum	1928
Range	3113
Interquartile range (IQR)	175

Descriptive statistics

Standard deviation	271.1646968
Coefficient of variation (CV)	-15.41249083
Kurtosis	5.535145492
Mean	-17.59382697
Median Absolute Deviation (MAD)	88
Skewness	1.153942158
Sum	-1128644
Variance	73530.2928
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
-39	260	0.4%
-35	250	0.4%
-25	247	0.4%
-23	245	0.4%
-41	241	0.4%
-30	239	0.4%
-19	238	0.4%
-42	238	0.4%
-51	237	0.4%
-49	235	0.4%
Other values (2286)	61720	96.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-1185	1	< 0.1%
-1155	1	< 0.1%
-1152	1	< 0.1%
-1137	1	< 0.1%
-1125	1	< 0.1%

Value	Count	Frequency (%)
1928	1	< 0.1%
1902	1	< 0.1%
1744	1	< 0.1%
1738	1	< 0.1%
1693	1	< 0.1%

UK_EURO
Real number (ℝ)

Distinct	1286
Distinct (%)	2.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	-452.9123928
Minimum	-926
Maximum	427
Zeros	10
Zeros (%)	< 0.1%
Memory size	501.3 KiB

Quantile statistics

Minimum	-926
5-th percentile	-814
Q1	-622
median	-463
Q3	-298
95-th percentile	-94
Maximum	427
Range	1353
Interquartile range (IQR)	324

Descriptive statistics

Standard deviation	228.3948625
Coefficient of variation (CV)	-0.5042804439
Kurtosis	0.04218481976
Mean	-452.9123928
Median Absolute Deviation (MAD)	162
Skewness	0.3595837202
Sum	-29054330
Variance	52164.21322
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
-472	139	0.2%
-480	138	0.2%
-482	138	0.2%
-353	138	0.2%
-476	137	0.2%
-346	136	0.2%
-478	136	0.2%
-468	135	0.2%
-486	134	0.2%
-344	134	0.2%
Other values (1276)	62785	97.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-926	1	< 0.1%
-925	1	< 0.1%
-922	1	< 0.1%
-917	1	< 0.1%
-916	1	< 0.1%

Value	Count	Frequency (%)
427	1	< 0.1%
421	1	< 0.1%
416	1	< 0.1%
411	1	< 0.1%
406	1	< 0.1%

UK_MLD
Real number (ℝ)

Distinct	744
Distinct (%)	1.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	-53.5909431
Minimum	-687
Maximum	377
Zeros	455
Zeros (%)	0.7%
Memory size	501.3 KiB

Quantile statistics

Minimum	-687
5-th percentile	-182
Q1	-95
median	-38
Q3	3
95-th percentile	48
Maximum	377
Range	1064
Interquartile range (IQR)	98

Descriptive statistics

Standard deviation	86.50529366
Coefficient of variation (CV)	-1.614177483
Kurtosis	7.454736773
Mean	-53.5909431
Median Absolute Deviation (MAD)	46
Skewness	-1.987228732
Sum	-3437859
Variance	7483.165832
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
-1	467	0.7%
2	457	0.7%
0	455	0.7%
-5	455	0.7%
6	454	0.7%
3	452	0.7%
-4	451	0.7%
-2	450	0.7%
5	446	0.7%
4	442	0.7%
Other values (734)	59621	92.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-687	1	< 0.1%
-672	1	< 0.1%
-671	1	< 0.1%
-653	1	< 0.1%
-642	1	< 0.1%

Value	Count	Frequency (%)
377	1	< 0.1%
314	1	< 0.1%
218	1	< 0.1%
192	1	< 0.1%
190	1	< 0.1%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

Count
Matrix

A simple visualization of nullity by column.

Nullity matrix is a data-dense display which lets you quickly visually pick out patterns in data completion.

First rows

	datereport	AES	TEC	VDE	TES	GES	GAES_GEN	CONSUMPTION	GAES_PUMP	UK_BLR_RUS	UK_EURO	UK_MLD
0	2021-04-28 07:00:00	10303.0	1126.0	630.0	3379.0	1902.0	0.0	16820.0	0.0	-42.0	-413.0	-65.0
1	2021-04-28 06:00:00	10304.0	963.0	396.0	3068.0	1098.0	0.0	15271.0	-87.0	-35.0	-444.0	8.0
2	2021-04-28 05:00:00	10263.0	938.0	386.0	3029.0	923.0	0.0	14498.0	-488.0	-86.0	-467.0	0.0
3	2021-04-28 04:00:00	10229.0	944.0	357.0	3037.0	1162.0	0.0	14514.0	-705.0	-84.0	-448.0	22.0
4	2021-04-28 03:00:00	10113.0	939.0	436.0	3045.0	1264.0	0.0	14468.0	-710.0	-94.0	-463.0	-62.0
5	2021-04-28 02:00:00	10250.0	977.0	406.0	2983.0	921.0	0.0	14650.0	-306.0	-86.0	-450.0	-45.0
6	2021-04-28 01:00:00	10242.0	1065.0	384.0	3183.0	918.0	0.0	15297.0	-13.0	-40.0	-448.0	6.0
7	2021-04-28 00:00:00	10273.0	1135.0	355.0	3747.0	930.0	40.0	16117.0	0.0	70.0	-400.0	-33.0
8	2021-04-27 23:00:00	10509.0	1255.0	322.0	3815.0	2147.0	630.0	17794.0	0.0	-568.0	-358.0	42.0
9	2021-04-27 22:00:00	10644.0	1260.0	375.0	3965.0	2238.0	562.0	18744.0	0.0	22.0	-300.0	-22.0

Last rows

	datereport	AES	TEC	VDE	TES	GES	CONSUMPTION	GAES_PUMP	UK_BLR_RUS	UK_EURO	UK_MLD
64140	2014-01-01 10:00:00	10455.0	2020.0	118.0	8303.0	862.0	19014.0	-88.0	-120.0	-405.0	-76.0
64141	2014-01-01 09:00:00	10479.0	2017.0	113.0	8233.0	557.0	18276.0	-480.0	-123.0	-402.0	-69.0
64142	2014-01-01 08:00:00	10493.0	2013.0	54.0	8179.0	72.0	17480.0	-703.0	-173.0	-399.0	-17.0
64143	2014-01-01 07:00:00	10469.0	2013.0	55.0	8202.0	193.0	17565.0	-707.0	-200.0	-410.0	-11.0
64144	2014-01-01 06:00:00	10473.0	2013.0	55.0	8201.0	434.0	17911.0	-707.0	-93.0	-416.0	-8.0
64145	2014-01-01 05:00:00	10427.0	2009.0	49.0	8355.0	274.0	18072.0	-488.0	-90.0	-421.0	-11.0
64146	2014-01-01 04:00:00	10475.0	2014.0	42.0	8369.0	185.0	18453.0	0.0	-175.0	-399.0	-26.0
64147	2014-01-01 03:00:00	10515.0	2022.0	54.0	8731.0	355.0	19107.0	0.0	-59.0	-400.0	-60.0
64148	2014-01-01 02:00:00	10606.0	2014.0	49.0	8885.0	863.0	19665.0	0.0	-200.0	-409.0	-104.0
64149	2014-01-01 01:00:00	10728.0	2016.0	50.0	8892.0	1809.0	20586.0	0.0	-365.0	-355.0	-150.0

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Missing values

Sample

First rows

Last rows